CNN-based surrogates have become prevalent in scientific applications to replace conventional time-consuming physical approaches. Although these surrogates can yield satisfactory results with significantly lower computation costs over small training datasets, our benchmarking results show that data-loading overhead becomes the major performance bottleneck when training surrogates with large datasets. In practice, surrogates are usually trained with high-resolution scientific data, which can easily reach the terabyte scale. Several state-of-the-art data loaders are proposed to improve the loading throughput in general CNN training; however, they are sub-optimal when applied to the surrogate training. In this work, we propose SOLAR, a surrogate data loader, that can ultimately increase loading throughput during the training. It leverages our three key observations during the benchmarking and contains three novel designs. Specifically, SOLAR first generates a pre-determined shuffled index list and accordingly optimizes the global access order and the buffer eviction scheme to maximize the data reuse and the buffer hit rate. It then proposes a tradeoff between lightweight computational imbalance and heavyweight loading workload imbalance to speed up the overall training. It finally optimizes its data access pattern with HDF5 to achieve a better parallel I/O throughput. Our evaluation with three scientific surrogates and 32 GPUs illustrates that SOLAR can achieve up to 24.4X speedup over PyTorch Data Loader and 3.52X speedup over state-of-the-art data loaders.

Traditionally, data analysis and theory have been viewed as separate disciplines, each feeding into fundamentally different types of models. Modern deep learning technology is beginning to unify these two disciplines and will produce a new class of predictively powerful space weather models that combine the physical insights gained by data and theory. We call on NASA to invest in the research and infrastructure necessary for the heliophysics' community to take advantage of these advances.

We present a differentiable formulation of rigid-body contact dynamics for objects and robots represented as compositions of convex primitives. Existing optimization-based approaches simulating contact between convex primitives rely on a bilevel formulation that separates collision detection and contact simulation. These approaches are unreliable in realistic contact simulation scenarios because isolating the collision detection problem introduces contact location non-uniqueness. Our approach combines contact simulation and collision detection into a unified single-level optimization problem. This disambiguates the collision detection problem in a physics-informed manner. Compared to previous differentiable simulation approaches, our formulation features improved simulation robustness and a reduction in computational complexity by more than an order of magnitude. We illustrate the contact and collision differentiability on a robotic manipulation task requiring optimization-through-contact. We provide a numerically efficient implementation of our formulation in the Julia language called Silico.jl.

Humans form mental images of 3D scenes to support counterfactual imagination, planning, and motor control. Our abilities to predict the appearance and affordance of the scene from previously unobserved viewpoints aid us in performing manipulation tasks (e.g., 6-DoF kitting) with a level of ease that is currently out of reach for existing robot learning frameworks. In this work, we aim to build artificial systems that can analogously plan actions on top of imagined images. To this end, we introduce Mental Imagery for Robotic Affordances (MIRA), an action reasoning framework that optimizes actions with novel-view synthesis and affordance prediction in the loop. Given a set of 2D RGB images, MIRA builds a consistent 3D scene representation, through which we synthesize novel orthographic views amenable to pixel-wise affordances prediction for action optimization. We illustrate how this optimization process enables us to generalize to unseen out-of-plane rotations for 6-DoF robotic manipulation tasks given a limited number of demonstrations, paving the way toward machines that autonomously learn to understand the world around them for planning actions.

Accurate and timely rain prediction is crucial for decision making and is also a challenging task. This paper presents a solution which won the 2 nd prize in the Weather4cast 2022 NeurIPS competition using 3D U-Nets and EarthFormers for 8-hour probabilistic rain prediction based on multi-band satellite images. The spatial context effect of the input satellite image has been deeply explored and optimal context range has been found. Based on the imbalanced rain distribution, we trained multiple models with different loss functions. To further improve the model performance, multi-model ensemble and threshold optimization were used to produce the final probabilistic rain prediction. Experiment results and leaderboard scores demonstrate that optimal spatial context, combined loss function, multi-model ensemble, and threshold optimization all provide modest model gain. A permutation test was used to analyze the effect of each satellite band on rain prediction, and results show that satellite bands signifying cloudtop phase (8.7 um) and cloud-top height (10.8 and 13.4 um) are the best predictors for rain prediction. The source code is available at https://github.com/bugsuse/weather4cast-2022-stage2.

已经证明，经过代码完成培训的大型语言模型（LLMS）能够合成DocStrings的简单Python程序[1]。我们发现这些代码编写的LLM可以被重新使用以编写机器人策略代码，给定自然语言命令。具体而言，策略代码可以表达处理感知输出的功能或反馈循环（例如，从对象检测器[2]，[3]）并参数化控制原始API。当作为输入提供了几个示例命令（格式为注释）后，然后是相应的策略代码（通过少量提示），LLMS可以接收新命令并自主重新编写API调用以分别生成新的策略代码。通过链接经典的逻辑结构并引用第三方库（例如，numpy，shapely）执行算术，以这种方式使用的LLM可以编写（i）（i）表现出空间几何推理的机器人策略，（ii）（ii）将其推广到新的说明和新指令和新指令和（iii）根据上下文（即行为常识）规定模棱两可的描述（例如“更快”）的精确值（例如，速度）。本文将代码作为策略介绍：语言模型生成程序的以机器人为中心的形式化（LMP），该程序可以代表反应性策略（例如阻抗控制器），以及基于Waypoint的策略（基于远见的选择，基于轨迹，基于轨迹，控制），在多个真实的机器人平台上展示。我们方法的核心是促使层次代码 - 代码（递归定义未定义的功能），该代码可以编写更复杂的代码，还可以改善最新的代码，以解决HOMANEVAL [1]基准中的39.8％的问题。代码和视频可从https://code-as-policies.github.io获得。

最近的作品表明，如何将大语言模型（LLM）的推理能力应用于自然语言处理以外的领域，例如机器人的计划和互动。这些具体的问题要求代理商了解世界上许多语义方面：可用技能的曲目，这些技能如何影响世界以及对世界的变化如何映射回该语言。在体现环境中规划的LLMS不仅需要考虑要做什么技能，还需要考虑如何以及何时进行操作 - 答案随着时间的推移而变化，以响应代理商自己的选择。在这项工作中，我们调查了在这种体现的环境中使用的LLM在多大程度上可以推论通过自然语言提供的反馈来源，而无需任何其他培训。我们建议，通过利用环境反馈，LLM能够形成内部独白，使他们能够在机器人控制方案中进行更丰富的处理和计划。我们研究了各种反馈来源，例如成功检测，场景描述和人类互动。我们发现，闭环语言反馈显着改善了三个领域的高级指导完成，包括模拟和真实的桌面顶部重新排列任务以及现实世界中厨房环境中的长途移动操作任务。

当代人工神经网络（ANN）是经过训练的端到端，共同学习功能和分类器以完成感兴趣的任务。尽管非常有效，但这种范式在组装带注释的特定任务数据集和培训大规模网络方面施加了巨大的成本。我们建议通过引入视觉生物标志物分类的辅助预任务来将特征从下游肺超声任务中学习。我们证明，通过培训模型来预测生物标记标签，可以从超声视频中学习一个内容丰富，简洁和可解释的功能空间。值得注意的是，可以从弱视频尺度监督注释的数据中培训生物标志物功能提取器。这些功能可以由针对各种临床任务的各种下游专家模型（诊断，肺严重程度，S/F比）使用。至关重要的是，特定于任务的专家模型的准确性与直接训练此类目标任务的端到端模型相当，同时训练成本大大降低。

机器学习传感器代表了嵌入式机器学习应用程序未来的范式转移。当前的嵌入式机器学习（ML）实例化遭受了复杂的整合，缺乏模块化以及数据流动的隐私和安全问题。本文提出了一个以数据为中心的范式，用于将传感器智能嵌入边缘设备上，以应对这些挑战。我们对“传感器2.0”的愿景需要将传感器输入数据和ML处理从硬件级别隔离到更广泛的系统，并提供一个薄的界面，以模拟传统传感器的功能。这种分离导致模块化且易于使用的ML传感器设备。我们讨论了将ML处理构建到嵌入式系统上控制微处理器的软件堆栈中的标准方法所带来的挑战，以及ML传感器的模块化如何减轻这些问题。 ML传感器提高了隐私和准确性，同时使系统构建者更容易将ML集成到其产品中，以简单的组件。我们提供了预期的ML传感器和说明性数据表的例子，以表现出来，并希望这将建立对话使我们朝着传感器2.0迈进。

可变形的对象操作需要与机器人感应方式兼容的计算有效表示。在本文中，我们提出了Virdo：可变形弹性对象的隐式，多模式和连续表示。Virdo直接在视觉（点云）和触觉（反作用力）方式上运行，并了解了接触位置和力量丰富的潜在嵌入，以预测受外部接触的物体变形。 - 具有密集无监督的对应关系的模式重建，ii）概括为看不见的接触地层，iii）抑制了局部粘膜反馈的状态估计